MultiAgentBench introduces a new benchmark and MARBLE framework for evaluating large language model-based multi-agent systems in both collaborative and com Overview MultiAgentBench is a new benchmark for evaluating LLM-based multi-agent systems Focuses on both collaborative and competitive scenarios with 2-10 agents Contains 8 distinct tasks across negotiation, gaming, and coordination domains Evaluates different LLM models including GPT-4, Claude, and Gemini Reveals limitations in complex multi-agent interactions, especially competitive. Modular Design: Easily extend or replace components like agents, environments, and LLM integrations. Multi-Agent Support: Model complex interactions between multiple agents with hierarchical or cooperative execution modes. LLM Integration: Interface with various LLM providers (OpenAI, etc.) through a unified API. Shared Memory: Implement shared memory mechanisms for agent communication and. Large Language Models (LLMs) have propelled the emergence of sophisticated Multi-Agent Systems (MAS) that leverage language-driven reasoning, collaboration, and autonomous decision-making. This paper presents a comprehensive review of state-of-the-art LLM-based frameworks for building MAS - including AutoGen, CrewAI, CAMEL, ChatDev, LangGraph, and Google DeepMind’s Agent Development Kit (ADK. They propose a benchmark to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. ACL 2025's MultiAgentBench tests 6 domains from collaborative research to adversarial Werewolf. Here's what the results reveal about frontier model performance when agents must coordinate or compete. The paper introduces MultiAgentBench, a benchmarking framework that evaluates LLM-based multi-agent collaboration and competition using novel metrics. Join the discussion on this paper page MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators.